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Integrating  Evaluation  into 
Tobacco  Programs  for  Asian  American 
and  Pacific  Islander  Communities 


This  evaluation  kit  is  designed  to  help  people  who  are  developing 
or  implementing  tobacco  control  programs  for  Asian  American 
and  Pacific  Islander  (AAPI)  communities. 

The  goals  of  this  kit  are: 

•  to  help  program  planners  understand  the  value  of  evaluation 
for  their  program 

•  to  identify  ways  to  plan  and  incorporate  evaluations 

•  to  provide  different  methods  for  conducting  evaluations 

•  to  provide  case  studies  of  how  evaluations  can  be  planned 
for  specific  AAPI  tobacco  control  programs. 

This  kit  is  divided  into: 

the  following  sections:  1)  a  question  and  answer  section  that  provides 
basic  evaluation  information,  2)  case  studies  of  specific  evalua¬ 
tion  planning  and  implementation  examples,  3)  a  checklist  of 
steps  to  plan  your  own  evaluation,  4)  a  worksheet  and  5)  a  list 
of  resources  for  further  information. 
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and  Answers 


O  What  is  evaluation  and  why  is  it  important  ? 
Evaluation  helps  you  to  determine  the  effects  of 
your  activity  or  program  on  the  participants  and  on 
the  community  and  to  document  the  successes  and 
challenges.  The  goal  of  evaluation  is  to  help  you 
(and  others)  to  understand  what  your  program  is 
actually  accomplishing.  For  example,  is  your  pro¬ 
gram  being  implemented  the  way  you  planned?  Is  it 
reaching  the  kinds  of  people  you  wanted?  Is  it 
achieving  the  kinds  of  short-term  changes  (like 
increased  knowledge  about  the  harm  of  tobacco  use, 
or  increased  skills  in  refusing  to  start  using  tobacco) 
and  long-term  changes  (such  as  tobacco  use  preven¬ 
tion  or  cessation)  that  you  expected?  Is  it  making  a 
difference  in  the  long  run?  Evaluation  can  help  you 
to  answer  these  kinds  of  questions  and  more. 
Evaluation  is  also  important  because  it  can  help  you 
to  make  changes  to  your  program  during  imple¬ 
mentation,  as  well  as  areas  for  improvement  in  the 
future.  For  instance,  evaluation  of  a  youth  tobacco 
use  prevention  program  could  identify  that  the  cur¬ 
rent  program  participants  were  not  those  who  are  at 
high  risk  for  smoking.  If  you  found  this  out  mid¬ 
way  through  your  program  implementation,  you 
could  change  your  program  outreach  to  reach  the 
kinds  of  youth  you  originally  intended. 

Finally,  evaluations  are  important  for  Asian 
American  and  Pacific  Islander  tobacco  control 


programs  because  they  help  us  to  identify  and 
document  what  works  for  our  communities.  There 
are  very  few  AAPI  tobacco  control  programs  that 
have  had  adequate  time  and  resources  to  plan  and 
conduct  extensive  evaluations.  While  this  is  a  reality 
of  community-based  programs,  the  lack  of  evaluation 
has  resulted  in  the  lack  of  proven  models  that  work. 
We  must  conduct  evaluations  of  our  own  programs 
in  order  to  document  efforts  and  measure  change. 

O  How  do  you  overcome  fear  of  evaluations  ? 

Most  often,  people  are  intimidated  by  evaluation 
either  because  they  don’t  know  what  it  is  for,  or 
they  do  not  know  how  to  do  it.  The  reasons  for 
doing  evaluation  are  to  benefit  the  program  and 
document  the  successes  and  challenges.  Program 
evaluations  are  less  concerned  about  whether  an 
individual  did  a  "good"  or  "poor"  job  (in  fact,  eval¬ 
uations  that  assess  job  performance  are  different 
from  program  evaluations).  However,  many  people 
do  not  understand  this  difference,  and  thus  believe 
that  the  purpose  of  evaluation  is  to  find  fault  with 
how  they  did  their  job. 

Since  people  may  be  intimidated  by  evaluations, 
they  believe  that  it  takes  a  lot  of  training  and 
expertise  to  execute.  Many  people,  however,  are 
already  doing  evaluation-type  activities  as  part  of 
their  program!  For  instance,  do  you  keep  docu- 


ments  like  sign-in  sheets,  meeting  notes,  photo¬ 
graphs  of  events  in  the  community,  or  newspaper 
clippings  of  articles  on  your  program?  These  docu¬ 
ments  can  all  be  used  to  evaluate  your  program!  A 
review  of  sign-in  sheets  of  youth  who  attended  your 
tobacco  prevention  education  sessions,  lor  example, 
could  help  you  to  understand  the  types  of  people 
who  were  involved  —  how  many,  what  ethnicity, 
what  gender,  what  age,  etc.  If  your  original  program 
goal  was  to  reach  middle  school  Cambodian  stu¬ 
dents,  a  review  of  your  sign-in  sheets  could  help  you 
to  determine  whether  you  actually  reached  those 
types  of  students. 

It  is  important  to  de-mystify  evaluation  for  everyone, 
in  order  to  show  them  that  evaluation  can  be  based 
on  common-sense  approaches  to  answering  basic 
questions  about  the  process  or  outcomes  of  a  program. 

O  What  are  the  different  types  of  evaluations  ? 
There  are  three  basic  types  of  evaluation:  process, 
outcome,  and  impact. 

•  Process  evaluation  tells  you  about  how  your  pro¬ 
gram  was  implemented.  Process  evaluation  looks 
at  what  your  program  did,  what  was  distributed  to 
the  community,  who  and  how  many  people  were 
involved,  when  the  event  took  place,  and  what 
they  were  involved  in.  For  instance,  the  example 
of  reviewing  sign-in  sheets  (in  the  previous  sec- 


(Case  #1 ) 

Using  Interviews  of  Youth  Leaders  to 
Evaluate  Program  Outcomes  in  the  Pacific 

On  an  island  in  the  Pacific,  local  governmental 
leaders  have  developed  a  successful  youth  for 
youth  leadership  program. The  long-term  goal 
of  the  program  is  to  prevent  youth  from  using 
tobacco,  including  not  smoking  or  chewing. 
The  program  recruits  middle  school  and  high 
school  students,  and  requires  them  to  sign  a 
pledge  to  not  use  tobacco  during  the  month 
of  their  training.  During  training,  teen  leaders 
train  other  teens  in  tobacco  control,  including 
health  effects  and  policies,  as  well  as  in  skills 
to  stay  tobacco  free. The  program  is  so  suc¬ 
cessful  that  it  has  become  a  model  for  other 
like  programs  across  the  Pacific  and  on  the 
U.S.The  major  evaluation  question  was 
whether  the  training  program  "worked"  to 
help  prevent  more  teens  from  not  smoking 
than  other  types  of  educational  programs.  In 
order  to  answer  this,  staff  decided  to  inter¬ 
view  six  randomly  identified  teen  participants 
each  year  at  the  beginning  of  their  training 
and  one  year  later.  Each  interview  took  about 
half  an  hour  to  complete,  and  asked  each  par¬ 
ticipant  three  questions:  I)  why  they  believe 
they  are  at  risk  for  tobacco  use,  2)  what  they 
have  learned  to  help  prevent  them  from  using 
tobacco,  and  3)  how  they  think  they  may  or 
may  not  be  able  to  stay  tobacco-free.  From 
these  interviews,  the  staff  learned  that  the 
more  at-risk  a  teen  feels  he  or  she  is,  the 
more  things  they  learned  from  the  program, 
and  the  more  they  believed  they  could  stay 
away  from  tobacco.These  interviews  are  con¬ 
tinuing  to  be  conducted,  sometimes  modified 
to  include  other  questions  of  importance  for 
that  specific  year. The  "findings"  from  these 
interviews  have  been  included  in  the  yearly 
reports  of  progress  to  the  Department  of 
Health  Services,  which  continues  to  give 
money  each  year  to  sustain  the  program. 


tion)  is  a  form  of  process  evaluation.  Process 
evaluation  is  often  the  most  frequently  used  type 
of  evaluation  that  is  conducted  for  a  program. 

•  Outcome  evaluation  looks  at  the  immediate  effects 
of  your  program,  such  as  changes  in  program  par¬ 
ticipants’  tobacco  use  knowledge,  attitudes  and 
behavior.  For  example,  post-test  surveys  of  youth 
who  attended  a  prevention  education  session  is  a 
form  of  outcome  evaluation.  Photographs  of  vendors 
who  change  the  placement  of  tobacco  advertisements 
and  promotional  items  could  be  another  way  to 
document  the  effects  of  a  business  education  pro¬ 
gram.  This  type  of  evaluation  is  very  important 
because  it  allows  you  to  understand  whether  your 
program  made  the  kind  of  difference  you  expected. 

•  Impact  evaluation  looks  at  the  long-term  effects  of 
our  program,  usually  on  the  reduction  in  disease 
and/or  deaths  due  to  tobacco  use  in  your  commu¬ 
nity.  This  type  of  evaluation  is  often  the  hardest  to 
do  because  it  often  takes  years  for  someone  who  uses 
tobacco  to  get  sick  or  die.  Most  likely,  your  program 
will  have  ended  long  before  we  know  whether  it 
saved  anyone’s  life.  However,  many  researchers  are 
interested  in  this  type  of  evaluation,  and  can  com¬ 
mit  the  time  needed  to  track  long-term  changes. 

For  the  purposes  of  this  tool  kit,  we  will  primarily 

focus  on  process  and  outcome  evaluation. 


O  How  do  I  start  to  plan  an  evaluation  of 
my  program  ? 

To  start  to  plan  a  program  evaluation,  you  should 
first  look  at  your  program’s  goals  and  objectives,  and 
answer  the  following  questions: 

O  Who  is  my  program  trying  to  reach? 

Prevention  education  programs  focusing  on  youth 
are  very  different  from  cessation  programs  focusing 
on  adults,  or  business  education  programs  focusing 
on  vendors.  The  answer  to  this  question  will  help 
you  to  eventually  determine  if  your  program 
reached  the  intended  audience. 

O  What  kinds  of  change  is  my  program  trying 
to  accomplish? 

If  you  program  is  successful,  what  would  your 
results  look  like?  The  answers  to  these  questions 
will  help  you  to  define  how  to  assess  the  outcomes 
ol  your  program. 

O  What  are  the  main  ways  that  my  program  tried 
to  change  the  community? 

Understanding  where  your  program  priorities  are 
will  help  you  to  tailor  the  evaluation  to  focus  on 
the  most  important  aspects,  rather  than  on  your 
entire  program. 


Evaluation  Methods, Advantages /Chal 
and  Resources  Needed 


Table  I 


Method 

Purpose 

Advantages  and  Challenges 

Resources  Needed 

Document 

review 

To  gather  informa¬ 
tion  from  pre-exist¬ 
ing  written  materials, 
such  as  meeting 
minutes,  reports, 
budgets,  newspaper 
articles,  etc. 

+  Most  common  method  for 
process  evaluation 

+  Good  method  for  reviewing 
materials  on  how  a  program 
was  implemented 

+  Uses  information  that  is 
already  available 

+  Can  be  done  by  program 
staff  themselves 

-  Takes  a  lot  of  time  to 
review  existing  files, 
especially  if  program  is 
complex  and  of  long 
duration 

Observation 

To  gather  informa¬ 
tion  on  how  the 

program  operates, 
or  on  visual  changes 
that  have  resulted 

+  Good  for  process  evaluation, 
because  you  can  get  an 
immediate  impression  about 
how  the  program  is  progressing 

—  Information  can  be  difficult 
to  interpret 

—  May  be  biased  by  how  staff 
interpret  the  information 

—  Events  that  are  seen  can  be 
time-dependent,  and  not 
representative  of  what  is 
really  going  on 

—  If  people  know  they  are 
being  observed,  they  may 
act  differently  than  usual 

+  Can  be  done  by  program 
staff  themselves 

-  Takes  a  lot  of  time  to 
review  existing  files, 
especially  if  program  is 
complex  and  of  long 
duration 

To  obtain  informa- 

+  Excellent  for  both  process 

-  Can  take  a  lot  of  time 

tion  from  individuals 

and  outcome  evaluation, 

to  conduct 

</i 

about  their  experi- 

because  you  can  get  in-depth 

-  May  require  data  inputting 

£ 

ences,  or  to  learn 

information  from  participants 

and  analysis  programs 

oj 

•  mm 

> 

more  about  their 

+  Can  ask  sensitive  questions 

-  May  require  assistance 

&- 

A\ 

answers  to  surveys 

that  require  confidentiality 

from  a  consultant  to 

-  Data  can  be  difficult  and 

design  the  interview 

c 

complex  to  analyze 

protocol  and  develop 

-  Interviewer  can  bias 

the  database  program 

the  responses 

KEY:  +  advantages 

-  disadvantages 


Table  I :  Evaluation  method,  advantages/challenges  and  resources  needed,  continued 


Method  Purpose  Advantages  and  Challenges 
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To  hold  discussions 
with  groups  of 
people  (10-12)  to 
understand  in-depth 
what  they  believe 
were  the  effects  of 
the  program,  or  how 
they  saw  the  program 
implemented. 


Good  for  outcome  evaluation, 
because  you  can  ask  people 
do  explain  how  the  program 
affected  them 


+ 


Can  identify  a  lot  of  issues 
and  effects 


Can  give  staff  better  under 
standing  of  the  program  from 
participants’  own  words 
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To  describe  a 
program  or 
experience  in  depth, 
often  using  some  or 
all  of  the  other 
evaluation  methods 
to  construct  a  case 
study. 


+ 


+ 
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To  get  information 
from  individual 
people  about  their 
changes  in  tobacco 
use  knowledge, 
attitudes,  and 
behavior. 
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+ 
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Good  for  combining  process 
and  outcome  evaluation 

Can  be  a  powerful  way  to 
describe  the  program 

Can  depict  personal  experiences, 
quotes,  and  unique  program 
processes 


Excellent  for  outcome 
evaluation 

Can  get  information  from  a 
lot  of  people 

Can  be  done  confidentially 
or  anonymously,  so  may  be 
more  valid 

Can  be  used  as  pre/post  tests 
to  measure  changes  from 
program  educational  sessions 
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+  Can  use  questions  from 
existing  surveys 

—  More  effective  when  using 
yes/no  or  true/false  type 
questions 


Resources  Needed 


Can  be  done  relatively 
quickly  (1-2  hours  per 
focus  group) 

Requires  a  good  facilitator 

Takes  time  to  analyze  and 
interpret  the  discussion 

May  require  extra  resources 
for  facilitator’s  time  and 
participant  incentives 


Can  take  a  lot  of  time 
to  collect  information, 
organize  and  describe 
the  program 


Takes  time  to  develop 
questionnaire 

May  require  database 
program  to  manage  and 
analyze  a  lot  of  data 

May  require  extra  resources 
for  participant  incentives 
and  data  analysis 


—  Are  more  impersonal  for 
participants,  and  usually  not 
good  for  getting  quotes 
in  participants’  own  words 


KEY:  +  advantages 

-  disadvantages 


O  What  evaluation  methods  work  best  if  I  don’t 
have  many  resources  ? 

There  are  many  methods  that  can  be  used  in  an 
evaluation,  some  require  more  resources  in  terms 
of  time  and  from  the  perspective  of  what  you  want 
to  find  out  —  some  methods  take  less  time  and 
money,  but  may  not  give  you  the  kind  of  informa¬ 
tion  you  need  to  answer  your  evaluation  questions. 
For  instance,  conducting  observations  of  a  teen 
leadership  education  program  and  writing  about  the 
outcomes  can  take  much  less  time  than  conducting 
a  survey  of  what  the  youth  participants’  believed 
were  their  outcomes. 

However,  the  observational  method  may  be  criti¬ 
cized  as  biased  (on  the  part  of  the  person  who  con¬ 
ducted  the  observations),  while  the  interviews  might 
be  considered  more  valid.  You  must  balance  consid¬ 
erations  of  resources  with  what  you  really  need  to 


know.  The  following  table  provides  an  overview  of 
the  different  kinds  of  evaluation  methods,  which 
are  loosely  ordered  from  more  simple  (and  requiring 
less  outside  resources)  to  more  complex. 

O  Do  all  evaluations  have  to  have  a  "before"  and 
"after"  design? 

Many  evaluators  believe  that  evaluations  must  be 
based  on  measures  of  what  was  going  on  before  the 
program,  and  what  happened  after  the  program.  A 
typical  example  is  the  use  of  pretests  and  posttests 
during  educational  sessions,  where  session  partici¬ 
pants  are  asked  to  complete  a  survey  at  the  start  of 
the  session,  followed  by  another  one  at  the  end  of 
the  session.  Changes  between  the  beginning  (i.e., 
pretest)  and  the  end  (i.e.,  posttest)are  assumed  to 
be  due  to  the  effects  of  the  education  itself.  A  more 
complex  example  is  the  use  of  baseline  measures 
(such  as  surveys)  at  the  start  of  your  program,  and 


Hypothetical  Case  Study  Example  of 
How  Evaluation  Can  Help! 

( Case  #2) 

Using  Program  Documents  to 
Develop  Case  Studies  of  an  Alternative 
Sponsorship  Education  Project 


A  community-based  organization  decided  to 
tackle  the  difficult  issue  of  educating  Asian 
American  and  Pacific  Islander  community  groups 
about  the  importance  of  refusing  money  from 
tobacco  companies. The  long-term  goal  was  to 
eliminate  all  tobacco  industry  money  support 
for  AAPI  community  groups  in  three  years, 
and  identify  alternative  non-tobacco  resources 
instead. Their  primary  evaluation  question  was 
“What  steps  should  programs  take  if  they  want 
to  conduct  tobacco  sponsorship  efforts  in  other 
areas?”  Staff  started  by  identifying  a  list  of 
community  groups  that  sponsored  annual  AAPI 
cultural  events  and  fairs,  like  the  Vietnamese 
Tet  festival,  Korean  fall  celebration,  and  Pacific 
Islander  festival.  About  one  year  before  each 
event,  the  staff  also  developed  an  in-depth  log 
file  for  each  community  group,  in  which  each 
staff  documented  the  dates,  times,  and  content 
of  conversations  with  the  group,  any  barriers 
and  concerns,  possible  alternative  funding 
sources  to  approach  (such  as  local  foundations 
that  could  give  $5000  to  print  a  festival  pro¬ 
gram,  instead  of  the  program,  being  printed  by 
a  tobacco  company),  and  ultimate  outcomes 
of  the  effort.These  conversations  with  local 
groups  occurred  over  the  course  of  the  year, 
and  by  the  end  the  staff  had  very  large  files  on 
how  they  worked  with  each  community  group. 
Finally,  on  the  day  of  the  community  event,  staff 
attended  and  documented  through  photographs 
the  presence  and  the  lack  of  tobacco  company 
advertisement  and  sponsorship.  During  the 
last  two  months  of  the  program,  staff  took 
all  the  information  that  they  collected  and 
constructed  "case  studies"  of  each  community 
group  (without  using  the  group’s  name  or  any 
specific  ethnic  community  identifiers  in  order 
to  maintain  confidentiality)  —  whether  or  not 
they  were  successful  in  changing  the  group’s 
sponsorship  acceptance.  These  case  studies 
were  then  distributed  to  all  agency  staff  to 
train  them  on  how  to  go  about  doing  anti¬ 
sponsorship  type  activities  for  the  future. 


then  repeating  the  measures  at  the  end  to  assess 
community-wide  changes.  [If  you  are  interested  in 
how  to  conduct  baseline  measures,  see  APPEAL’S 
publication,  "Conducting  Needs  Assessments  lor 
Tobacco  Control  in  Asian  American  and  Pacific 
Islander  Communities."] 

However,  it  is  often  difficult  to  get  information  from 
participants  or  from  the  entire  community  before 
the  start  of  the  program.  Think  about  the  example 
of  a  restaurant  education  program  in  the  Cambodian 
community  —  it  would  be  very  difficult  to  count 
all  of  the  restaurants  that  support  anti-smoking  laws 
before  your  program  began.  But  in  this  example,  it 
would  be  possible  for  people  to  record  how  many 
restaurants  they  spoke  to  throughout  the  program, 
how  many  were  initially  resistant  to  a  new  law 
(e.g.,  believed  that  anti-smoking  laws  would  harm 
their  business),  and  how  many  were  willing  to 
change  to  support  a  new  law  alter  the  stalf  talked 
to  them.  This  kind  of  documentation  of  the  pro¬ 
gram  uses  both  "observation"  and  "interview" 
methods  of  evaluation,  which  could  be  used  to 
develop  case  studies  of  program  successes.  In  this 
case,  the  "power"  of  evaluation  is  to  document  the 
kinds  of  things  that  stall  discussed  with  the  restau¬ 
rant  owners,  the  kinds  of  barriers  to  any  new  laws, 


and  the  kinds  of  information  that  helped  convince 
owners  of  the  importance  of  a  new  law. 

Therefore,  there  is  no  hard  and  fast  rule  about  need¬ 
ing  "before"  and  "after"  designs  for  your  evaluation. 

O  What  resources  are  available  or  who  can  help 
with  our  evaluation? 

In  general,  there  are  also  some  general  rules  you  may 
want  to  follow  to  maximize  the  resources  you  have 
for  an  evaluation: 

•  First,  start  with  an  assessment  of  the  capacities  of 
your  staff.  Someone  who  is  a  good  listener  might 
make  an  effective  interviewer.  Someone  who  likes 
to  take  photos  and  document  events  might  make 
a  good  observer.  Someone  who  knows  programs 
like  Microsoft  Excel  or  Access  might  be  able  to 
design  a  database  program  for  survey  data. 

•  Think  about  how  many  resources  you  have  avail¬ 
able  to  evaluate  your  program.  A  basic  rule  is  to 
allocate  about  10%  of  your  program’s  budget  to 
evaluation  activities.  These  resources  may  be  more 
or  less  —  just  like  when  planning  a  program,  you 
need  to  think  about  who  will  be  responsible  for 
planning,  designing,  conducting,  and  compiling 
the  evaluation  information. 


•  Also,  good  evaluations  include  incentives  for  com¬ 
munity  members  to  participate.  If  you  are  planning 
to  do  a  focus  group  or  survey,  you  should  include 
resources  for  "thank  you"  incentive  items,  trans¬ 
portation,  etc.  If  your  evaluation  will  be  based 
upon  observations,  you  can  enlist  youth  to  help 
you  take  pictures  of  what  they  see  has  changed  in 
their  community  (such  as  less  tobacco  promotional 
items  at  their  local  stores  or  restaurants).  You 
should  also  provide  participants  with  the  final 
evaluation  report,  or  at  minimum,  a  summary  of 
the  findings. 

•  Finally,  do  you  have  a  university  nearby  from  which 
you  can  recruit  students,  faculty  or  researchers  to 
assist  you  with  your  evaluation?  Oftentimes  they 
may  not  understand  the  complex  issues  of  tobacco 
control  for  AAPIs  —  in  other  words,  they  may 
not  have  the  cultural  competence  to  develop  an 
entire  evaluation  plan.  However,  they  might  have 
specific  skills  that  can  help  you  conduct  your  own 
evaluation  —  skills  like  putting  questions  into  a 
survey  format,  designing  a  database  program,  and 
analyzing  data  for  presentations  or  reports. 


How  to  Integrate  Evaluation 

-  into  Tobacco 

Control  for  AAPIs 


Now  that  you  have  a  basic  background  on  the  value 
of  evaluations  for  your  tobacco  control  program, 
here  is  a  checklist  of  steps  that  you  can  take  to 
incorporate  evaluation  into  your  program.  The 
evaluation  steps  you  take  do  not  necessarily  have 
to  occur  in  this  order,  but  this  checklist  should  help 
you  to  get  started  in  your  effort! 


The  Twelve  Steps  to  Evaluation: 


Step  1:  Identify  and  describe  the  community  you 
are  working  with. 

You  probably  already  have  a  good  understanding 
of  your  community,  and  so  may  not  be  used  to 
describing  the  community  to  others.  But  in  order 
for  an  evaluation  to  be  helpful  to  people  outside  of 
your  community,  they  must  understand  the  unique 
factors  that  influence  tobacco  use.  Such  factors  may 
include  the  community’s  size,  ethnicity,  geographic 
location,  age,  immigration  status  (if  applicable),  and 
general  tobacco  use  characteristics. 

Step  2:  Identify  what  your  activity  or  program  is 
trying  to  achieve. 

A  good  place  to  start  is  with  your  program’s  goals 
and  objectives  —  what  are  you  trying  to  change  in 
the  community,  with  whom,  and  by  when?  If  you 
do  not  have  a  work  plan  yet,  what  do  you  think  are 
the  activities  or  steps  that  need  to  be  addressed? 

Step  3:  Identify  what  activities  your  program  is,  or 
will  be,  doing  in  the  community. 

In  order  to  decide  which  evaluation  method(s)  to 
use,  you  need  to  first  understand  what  activities 
your  program  will  be  implementing.  In  the 
example  on  page  3,  the  "activities"  were  staff 
working  closely  with  members  from  other  commu¬ 
nity  groups  to  educate  them  about  tobacco  sponsor¬ 
ship,  identify  alternative  resources,  and  assist  them 
in  accessing  these  resources  while  refusing  tobacco 
sponsorship.  Since  the  activities  were  tailored  to 
each  community  group,  the  evaluation  "method" 
most  appropriate  was  document  review,  where  the 
"documents"  were  staff  notes  about  what  they  did 
each  time  they  spoke  with  a  community  group. 

You  should  tailor  your  methods  to  your  program 
activities  in  order  to  maximize  the  "fit"  between 
programs  and  evaluations. 


Step  4:  Determine  what  you  already  know  about 
your  program. 

Oftentimes,  staff  already  know  many  things  about 
the  process  or  outcomes  of  their  program.  For 
instance,  there  are  tobacco  control  staff  that  regularly 
outreach  to  families  in  their  homes  and  spend  time 
talking  with  them  about  issues  beyond  second  hand 
smoking.  For  these  staff,  they  can  often  identify 
the  ways  that  tobacco  use  in  the  community  has 
changed  just  from  observing  and  discussing  issues 
with  these  clients.  Together,  you  should  identify 
what  evaluation  questions  still  need  to  be  answered. 

Step  5:  Determine  what  questions  you  would  like 
an  evaluation  to  answer. 

Your  evaluation  should  focus  on  the  two  or  three 
most  important  questions  that  you  would  like  to 
address,  although  there  may  be  many  more  that 
you  would  like  to  tackle.  These  questions  should 
pertain  to  either  process  or  outcome  or  both 
(impact  evaluation,  if  you  remember,  is  difficult 
and  usually  requires  greater  expertise  to  conduct). 
You  should  try  to  make  sure  that  the  questions  are 
the  same  for  other  staff  and  for  the  community  — 
staff  and  community  members  are  much  more  like¬ 
ly  to  participate  in  an  evaluation  if  they  believe 
that  the  questions  you  are  answering  are  the  most 
important  to  address. 


I  »  ^  V  EL  Vi  V  I  n 


Step  6:  Determine  what  evaluation  method  you 
would  like  to  use  to  answer  the  question(s). 

Look  at  the  evaluation  questions  you  have  developed 
and  ask  yourself  three  questions: 

1 )  Who  do  you  need  to  talk  to  in  order  to  answer 
the  question?  2)  How  detailed  should  the  informa¬ 
tion  be  that  you  are  collecting?  3)  How  much  time 
do  you  have  to  collect  and  analyze  the  information? 
Based  on  the  answers  to  these  questions,  you  should 
be  able  to  identify  at  least  one  or  two  methods  from 
Table  1  that  can  be  adapted  to  suit  your  needs.  If 
you  have  never  done  an  evaluation  before,  you  may 
want  to  select  a  more  simple  evaluation  method 
(that  appears  earlier  in  Table  1)  in  order  to  build 
your  skills  to  conduct  more  complex  evaluation 
methods  in  the  future. 

Step  7:  Identify  available  resources  to  conduct 
the  evaluation. 

The  most  important  resource  is  your  own  program 
staff,  because  they  are  the  ones  who  will  be  most 
interested  and  invested  in  what  the  evaluation 
uncovers.  Other  common  resource  needs  may 
include  incentives  for  program  participants,  tape 


recorders  and  transcribers  (for  focus  groups  or  inter¬ 
views),  consultants  for  facilitation  (for  focus  groups) 
or  database  management  and  analysis  (for  surveys 
and  interviews),  survey  printing,  and/or  disposable 
cameras  and  photo  development  (for  observations). 

Step  8:  Identify  tools  that  can  help  with 
the  evaluation. 

Often  there  are  tools  other  people  have  developed 
to  conduct  their  evaluations.  Pre-existing  surveys  are 
the  most  common  tool,  since  many  people  have 
conducted  pretests  or  surveys  for  their  populations 
(such  as  those  developed  by  the  Vietnamese 
Community  Health  Promotion  Project).  You  should 
not  be  shy  about  asking  others  for  tools  that  they 
have  used,  especially  since  this  can  save  you  a  lot  of 
time  and  resources. 

Step  9:  Collect  evaluation  data. 

The  best-planned  evaluations  begin  collecting  data 
at  the  same  time  that  the  program  itself  is  begin¬ 
ning.  This  means  that  data  collection  will  continue 
throughout  the  life  of  the  program,  and  will  be 
completely  integrated  into  the  program’s  activities. 
However,  it  is  common  for  evaluation  data  to  be 
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collected  closer  to  the  end  of  a  program,  when  staff 
have  more  time  to  devote  to  evaluation  planning 
and  collection.  In  this  case,  data  collection  can  be 
done  in  a  few  intensive  months  of  work.  However, 
a  weakness  in  waiting  to  collect  data  may  result  in 
lack  of  access  to  the  data  you  need.  Either  way, 
allow  yourself  and  your  staff  enough  time  to  con¬ 
duct  evaluation  activities  —  the  "  1 0%"  of  resources 
rule  means  that  10%  of  staff  time  should  also  be 
devoted  to  data  collection  over  the  entire  project 
time  period. 

Step  10:  Regularly  "analyze"  the  data  to  determine 
if  your  program  is  "on  course"  or  "needs  to 
make  adjustment.  " 

At  regular  times,  plan  to  analyze  the  evaluation  data 
to  see  if  the  kinds  of  information  you  are  getting 
are  answering  the  questions  you  have.  Relatively 
quicker  data  analysis  should  be  done  to  see  if  the 
program  is  "on  course,"  which  means  that  the  pro¬ 
gram  is  reaching  the  appropriate  people,  is  being 
implemented  the  way  planned,  and  is  resulting  in 
the  kinds  of  short-term  changes  you  expected.  More 
in-depth  analysis  should  look  at  all  the  data  to  more 
definitively  answer  your  evaluation  questions. 


Step  11:  Incorporate  evaluation  findings  into 
program  for  improvement  and  continuation. 

The  most  common  mistake  in  evaluations  is  to  not 
use  the  results  to  improve  the  program  and/or  plan 
for  future  programs.  Too  often,  evaluation  "reports" 
are  written  but  just  sit  on  a  bookshelf  gathering 
dust.  Focus  your  evaluation  on  getting  the  most 
"useable"  data  possible  —  and  let  the  academics 
write  the  elaborate  reports  that  the  typical  lay 
person  doesn’t  read.  The  most  important  audience 
for  your  evaluation  may  be  your  staff  and  your 
community,  so  make  sure  to  share  the  findings  with 
both  audiences  to  learn  how  to  improve  tobacco 
control  for  AAPI  communities! 

Step  12:  Disseminate  your  results  to  wider  audiences. 

We  also  make  the  mistake  of  not  sharing  our  suc¬ 
cesses  and  challenges  with  other  AAPI  tobacco 
control  advocates  outside  of  our  own  agencies. 

Take  the  time  to  share  your  results  at  conferences, 
and  through  list  serves,  or  publish  the  results 
in  newspapers,  tobacco  control  newsletters  and 
journals  in  order  to  tell  others  about  the  lessons 
you’ve  learned. 
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Additional  Resources 


There  are  very  few  evaluations  resources  that  are  available  and 
applicable  to  AAPI  tobacco  control  issues  and  needs.  The  fol¬ 
lowing  are  more  general  evaluation  tools  and  information  that 
you  can  also  refer  to  when  planning  your  evaluation.  But  the 
most  important  resource  is  yourself  and  your  staff  —  use  the 
"local  knowledge"  of  your  community  to  develop  and  imple¬ 
ment  an  evaluation  that  meets  your  needs  and  answers  your 
own  questions! 

The  Community  Toolbox:  Tools  for  Program  Evaluation  —  this 
is  a  website  designed  by  the  University  of  Kansas  and  offered 
free  to  the  public.  The  website  is:  http://ctbdsi.ukans.edu/ 

Fetterman,  David  M.  Foundations  of  Empowerment  Evaluation. 
Sage  Publications,  2000. 

Patton,  Michael  Quinn.  How  to  Use  Qualitative  Methods  in 
Evaluation,  2nd  Edition.  Sage  Publications,  1988. 

Albright  A,  Howard-Pitney  B,  Robert  S  and  Zicarelli  J.  Tell  Your 
Story:  Guidelines  for  Preparing  an  Evaluation  Report.  Sacramento, 
CA,  California  Department  of  Health  Services,  1 988 
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Notes: 


Have  questions?  Call,  write,  or  e-mail  us  at: 

APPEAL  439  -  23rd  Street,  Oakland,  CA  94612 

Phone:  (510)  272-9536,  lax:  (510)  272-0817,  e-mail:  info@aapcho.org 

Also  check  APPEAL’S  web  page  at  www.appealforcommunities.org 


Asian  Pacific  Partners  for  Empowerment  and  Leadership  (APPEAL)  is  a  national  network  of 
individuals  and  organizations  committed  to  working  towards  a  tobacco-free  Asian  American  and 
Pacific  Islander  (AAPI)  community.  APPEAL’S  mission  is  to  prevent  tobacco  use  and  improve  the 
health  status  in  the  AAPI  community  through  network  development,  capacity  building,  education, 
advocacy,  and  leadership.  APPEAL  is  currently  a  project  ol  AAPCHO. 
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